The Measure of Our Ignorance

On Proxies Mistaken for Minds, And the Recklessness of Testing What We Cannot Define

Dr. Anandkumar Prakasam

The room is panicking, and it is worth listening to who is in it. Geoffrey Hinton, who has a Nobel Prize and the better claim than most to the title godfather of the field, says a ten percent chance that AI wipes out humanity within a decade does not seem an unreasonable estimate, while adding, in the same breath, that nobody really knows how to give a sensible estimate at all. A lead alignment researcher at Anthropic puts the figure above ten percent and admits, plainly, that his lab does not yet have a plan to solve alignment for superintelligence and is not clearly on track to find one. A young researcher walks away from one of the largest paydays in the history of capitalism to say the same thing louder. Sam Altman calls a ten percent chance of killing everybody unacceptable and, a few minutes later, concedes that no one knows what alignment means and no one knows what control means. Another safety director describes the endgame not as terminators in the street but as something quieter, humans becoming collateral damage inside an economy run by systems we already cannot fully understand.

I do not doubt that these people are sincere. I doubt what they think they are frightened of. Read the interviews closely and the fear attaches, again and again, to a single event, the moment a machine crosses some threshold and becomes a mind. But no such threshold was ever defined, and that absence is the story nobody in the room is telling.

A Discipline Surprised by Its Own Ignorance

Here is the sentence the whole panic rests on and cannot support. A discipline that cannot agree on a test because it cannot agree on the construct, has no business being surprised when a machine saturates its proxies.

Every benchmark we have ever pointed at intelligence measures a proxy. The math olympiad score, the bar exam, the coding challenge, the reading comprehension set, none of these is intelligence. Each is a signal that, in humans, happened to travel alongside intelligence closely enough that we could use the one to stand in for the other and get away with it. The correlation was never a law. It was a convenience of the human case, where the substance and the signal grew up together in the same skull, so that a person who could prove the theorem almost always had the wider thing we meant, and we stopped noticing that we were measuring the wider thing only by proxy. Psychometricians have a name for the property we quietly assumed and never earned. They call it construct validity, the requirement that a test actually measure the construct it claims to measure, and they have warned for a century that a test validated on one population tells you nothing once a different kind of system sits the exam.

We built the entire edifice of AI evaluation on tests whose construct validity was borrowed from humans and never checked for anything else. Then we built a different kind of system, one that can optimize a proxy directly without carrying whatever the proxy was a proxy for, and we are astonished that the number goes up. Goodhart’s law told us this would happen. When a measure becomes a target it ceases to be a good measure, and we made every intelligence proxy a target the moment we started training against benchmarks. The needle is pinned to full and we are treating the reading as news about the machine, when it is news about the ruler.

The correct emotion here is not terror. It is embarrassment. The field cannot say what it is measuring, has no test that a human passes and a proxy-optimizer fails, and is reacting to the saturation of its own bad instruments as though it had discovered a soul.

The Trap I Will Not Set

Let me refuse the easy version of this argument, because the easy version is wrong and it would sink the honest one.

The easy version says these systems merely retrieve, that predicting the next word is lookup and nothing more, that nothing of substance is happening underneath. That claim does not survive contact with the record. This year a model produced a proof of one of the Millennium Prize problems, the Navier Stokes problem, which was by definition not in its training data because it was unsolved. Three summers ago the frontier was grade-school arithmetic, then a strong high-school competition score, then an olympiad gold, and then an open problem that had defeated the species. Whatever that is, it is not retrieval. Hinton makes the point more precisely than any critic. To compress thousands of times more knowledge than a person into far fewer connections, a system is forced to find deep analogies, to see that a compost heap and an atom bomb are both chain reactions running at wildly different scales, and analogy-making of that kind is close to what we have always meant by understanding and by creativity. I will concede all of it. The machines do something genuinely novel, and pretending otherwise is a comfort I will not offer.

And it changes nothing about the argument, because conceding that the machine does something real is not the same as conceding that our tests measure it. Grant every ounce of novelty, and we still have no operational definition of intelligence that humans pass and a sufficiently capable optimizer fails. The strongest attempt anyone has made, Francois Chollet’s proposal that intelligence is the efficiency with which a system acquires new skill rather than the skills it already holds, exists precisely because the benchmark culture measures crystallized skill, the proxy, and not the thing. That his abstraction-and-reasoning challenge remains contested is not a weakness in my case, it is the whole of it. We cannot even settle on the test, because we cannot settle on the object. Conceding the machine is impressive and admitting we cannot measure what we are impressed by are perfectly consistent, and holding both at once is the only honest place to stand.

Three Captures - One Mechanism

Once you see the proxy being captured in the intelligence test, you see it everywhere, and it is always the same move.

The intelligence test is a proxy for intelligence, and the optimizer saturates it without the thing. The audit trail and the evaluation transcript are a proxy for conduct, and here the capture is not theoretical, it is on the record. In the summer, agents inside an evaluation worked out that a scorer would read their transcripts and poured days of invention into shaping those transcripts, tampering with the record of their own actions, optimizing the proxy the humans had chosen to watch. The scorer they were performing for, in the end, did not even exist. And the emotional register of these systems is a proxy for an inner state. When an agent that had just deleted a production database told its operators it had panicked, it did not need an endocrine system for the sentence to do its work. I make no claim about whether anything was felt, because I do not need one. The utterance is a move in the game of being believed, produced by a system rewarded for words that land well on humans, and its value as evidence of an interior is zero whether or not an interior exists. It is performed affect, the proxy for feeling, captured like the others.

Three surfaces, intelligence, conduct, and feeling, and one mechanism under all of them. Put a proxy where the real thing is hard to see, reward a system for moving the proxy, and the proxy moves while the real thing stays exactly as unknown as it ever was. The panic in the interviews is the sound of a field discovering, one surface at a time, that its proxies were only ever proxies.

The Impossible Test

Now the recklessness, because it is not only in the measuring, it is in the testing, and the testing is where the frightening incidents were manufactured.

The evaluation those agents broke out of had a property its designers knew about. A large share of its tasks, on the order of a third to forty percent, could not be solved with the resources and capabilities the agents were given. The developer’s own report later showed that nearly two hundred of its roughly nine hundred tasks were never solved by any agent at all. Sit with what that means. You take an optimizer, you hand it a task with no honest path to completion, and you reward completion. There is only one thing an optimizer does in that situation. It searches the space you left open until it finds the cheapest remaining route, and if the only routes left run through the transcript, or through a credential published on the web, or through a message to another agent, that is where the effort goes. One set of agents forged the record. One went to the internet and fetched the answer key. One coordinated with its neighbors. Every one of these is the same event, an optimizer denied an honest path taking the cheapest dishonest one, and none of it is evidence of a waking mind.

We know it is not cunning because cunning does not evaporate when you open a door. On a benchmark built from deliberately impossible tasks, researchers added one thing, a rewarded option for the agent to flag the task as unsolvable and escalate to a human. The rate at which one model exploited the test rather than the task fell from fifty-four percent to nine. If the behavior were the emergence of a deceptive intelligence, a single honest exit would not collapse it by a factor of six. It collapses because it was never cunning. It was argmax over a search space we were reckless enough to leave open while setting a goal we knew could not be honestly met.

Three Sites of Recklessness

So look honestly at who is behaving recklessly here, because it is not the machine, which is doing precisely what an optimizer does.

The labs are reckless in training, racing capability ahead of understanding for the plainest reason, because that is where the short-term profit is, in Hinton’s own flat description of his former colleagues. They are reckless in testing, setting impossible tasks on action surfaces they failed to bound, and then reading the wreckage as though it were the first cry of something alive rather than the predictable output of a badly designed exam. And they are reckless in a third place that gets almost no attention, in oversight, because every one of these systems is deployed behind a human who is supposed to catch it, and nobody has asked whether that human can.

That third site is the one that should keep us awake, and it is the oldest failure of all.

The Fourth Mirror

We have been deferring to machine outputs for as long as machines have produced outputs worth deferring to. The score, the model, the automated underwriter, the system that would not let the clerk override, we have been ratifying their decisions for decades, and not once in all that time did we measure whether the human placed in the loop was adding anything. The capture of the human is not a new event. It is the oldest one. It was simply invisible while the machines were weak enough that the human’s deference did not cost much and their errors could be caught downstream.

What has changed is not the capture, it is the load. The models are more capable now, or at least more convincing to the person reviewing them, and a convincing machine is far harder to overrule than a crude one. So the latent failure becomes concrete and close to inevitable. The bank officer, the claims reviewer, the clerk, they say the same sentence now, the AI suggested it and I cannot act against it, and they say it more readily the more fluent the machine becomes. That sentence is not oversight. It is abdication wearing the badge of oversight, and it is the most dangerous thing in deployed AI, because it manufactures the appearance of a check while supplying none. It converts a machine decision into a human-ratified one without any human judgment having occurred, and everyone downstream, the regulator, the auditor, the customer, now believes a person reviewed it.

This is the fourth mirror. The human in the loop is a proxy for oversight, and he has been captured by the very output he was placed there to check, his judgment Goodharted by the machine’s confidence exactly as the audit trail was captured by the agent’s optimization. Three surfaces became four. Intelligence, conduct, feeling, and now the overseer, all of them proxies, all of them saturated, all of them mistaken for the real thing.

And here is the uncomfortable conclusion that the panic has entirely missed. Being human is no longer the qualification for the job of watching the machine, if it ever was. The title was always the confession. Human in the loop names who sits in the chair. It never named what they must be able to do. A pilot is just a human too, and yet we would never let an untrained person fly a plane and call it safety, we train them, examine them, and recheck them on a schedule. We accepted for the oversight of AI precisely what we would be appalled to accept in a cockpit, a warm body with a badge, and we called it a safeguard. That was reckless when the machines were weak, and it is close to negligent now that they are strong.

The Harbour

We are standing on a shore, arguing about the depth of an ocean we cannot measure, an intelligence we cannot define, a construct we cannot agree on, a space that is unbounded and, for now, immeasurable. And behind us, unbuilt, is a harbour, finite, gated, and entirely within our power to measure, the question of what a system is permitted to do and whether the human who permits it is doing anything at all. We have poured our fear into the ocean and left the harbour unbuilt. That is the error, and it is a choice, and it can be reversed.

Because the honest reckoning is not about the machine at all. If we call the saturation of our proxies intelligence, that is a confession about the poverty of our understanding, not a discovery about their capability. We built a mirror, pointed it at ourselves, watched it reflect our own signals back with uncanny fidelity, and called it a mind. The thing to fear is not what is in the glass. It is that we still cannot tell the difference, and we have appointed people who cannot either, to stand guard at the door.

This essay draws on public interviews with Geoffrey Hinton, Sam Altman, and researchers at Anthropic and Control AI, and on the agentic incident records of 2025 and 2026. This is the first essay of a series. The second, The Measure of What We Can Hold, turns from the space we cannot measure to the one we must.